Method and apparatus for processing natural language with a hierarchy of mapping routines

ABSTRACT

A method and apparatus for processing natural language and deducing meaning from a natural language input characterized by the steps of (a) receiving an ordered string of word objects having a natural language meaning, (b) selecting a word window length, and (c) successively moving the word window along the ordered string and analyzing the meaning of a substring of word objects that fall within the word window. The substring is removed from the ordered string if the substring has a recognized meaning, until all substrings of the ordered string that fit within the window have been analyzed. In a step (d), the word window length is reduced and step (c) is repeated until only an unrecognized residual of the ordered string remains. The meaning of the substring is analyzed by mapping the substring against a database using one or more mapping routines. The mapping routines are preferably arranged in a hierarchy, wherein a successive mapping routine is used to analyze the substring when a previous mapping routine in the hierarchy cannot map the substring. A computer-implemented task is determined from the recognized substrings and performed by the computer system. The apparatus of the present invention implements the method on a pen-based computer system, and the ordered string is preferably received from strokes entered by a stylus on a display screen of the pen-based computer or from a microphone receiving speech input.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional of application Ser. No. 08/215,064filed on Mar. 18, 1994, now U.S. Pat. No. 5,434,777, which is acontinuation-in-part of parent patent applications Ser. No. 07/889,225,filed May 27, 1992 on behalf of Luciw et al., entitled, "Method andApparatus for Deducing User Intent and Providing Computer ImplementedServices," now U.S. Pat. No. 5,390,281, and Ser. No. 08/099,861, filedJul. 30, 1993 on behalf of Luciw et al., entitled, "Method and Apparatusfor Providing Computer-Implemented Assistance," now U.S. Pat. No.5,477,477 which are hereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

The present invention relates generally to computer systems, and moreparticularly to methods for processing natural language input from usersof computer systems.

Computerized personal organizers are becoming increasingly popular. Theyperform such functions as keeping a calendar, an address book, a to-dolist, etc. While these functions can be provided by conventionalcomputer systems, they are more conveniently provided by personalorganizers which are relatively inexpensive, small, and lightweight(i.e. portable). Personal organizers are available from such companiesas Sharp and Casio of Japan.

A relatively new form of computer, the pen-based computer system, holdsforth the promise of a marriage of the power of a general purposecomputer with the functionality and small size of a personal organizer.A pen-based computer system is typically a small, hand-held computerwhere the primary method for inputting data includes a "pen" or stylus.A pen-based computer system is commonly housed in a generallyrectangular enclosure, and has a dual-function display assemblyproviding a viewing screen along one of the planar sides of theenclosure. The dual-function display assembly serves as both an inputdevice and an output device. When operating as an input device, thedisplay assembly senses the position of the tip of a stylus on theviewing screen and provides this positional information to thecomputer's central processing unit (CPU). Some display assemblies canalso sense the pressure of the stylus on the screen to provide furtherinformation to the CPU. When operating as an output device, the displayassembly presents computer-generated images on the screen.

The dual-function displays of pen-based computer systems permit users tooperate the computers as computerized notepads. For example, graphicalimages can be input into the pen-based computer by merely moving thestylus across the surface of the screen. As the CPU senses the positionand movement of the stylus, it generates a corresponding image on thescreen to create the illusion that the stylus is drawing the imagedirectly upon the screen, i.e. that the stylus is "inking" an image onthe screen. With suitable recognition software, the "ink" can beidentified as text and numeric information.

Computer systems, including pen-based computer systems, can alsoincorporate speech recognition. Although still mostly in thedevelopmental stage, speech recognition has many potential uses incomputer systems. For example, in a portable pen-based computer, speechinput and recognition can be used to conveniently record dictatedmessages or other spoken documents and to present the dictated words asan electronic text document for editing, printing, faxing, etc. or toinput commands into the computer system.

Stylus, speech, keyboard, and other forms of input can take the form of"natural language," i.e. an "utterance." Natural language is written orspoken input that is in a natural form that a person would use as ifspeaking or writing to another person. Non-natural language would belanguage, such as computer commands or programs, which is constrained bya limited syntax, structure, and/or scope.

Natural language input permits a user to easily interface with acomputer system using well-known language. One area in computer systemswhich is particularly well-adapted for natural language input iscomputerized assistance. Computers systems that use computerizedassistance deduce or hypothesize a user's intent and automaticallyprovide a service based on the deduced intent. For example, a tasks suchas printing a document or scheduling an appointment can be implementedby a computer system by deducing information from a user's inputs orpreferences. Natural language input allows a user to quickly specify aservice or task to be implemented using a complex form and thus is wellsuited for use with computerized assistance.

The recognition of natural language by computer systems has beendifficult due to the complex and shifting "rules" of human naturallanguage. A user may input a natural language phrase, such as, "meetingwith Bill on Tuesday at lunch" to instruct the computer system toschedule the meeting in a calendar application program. The computersystem typically recognizes the individual words to recognize themeaning of the entire phrase. However, by recognizing small portions ofthe input phrase separately, a complete, overall meaning for the phraseis often incorrect, inaccurate, or incomplete.

Natural language input has been typically processed using one of severalmethods. A natural phrase or string can be matched to a known phrase ina phrase lexicon used by the computer. A long command is typicallyprocessed by matching individual words or smaller phrases to items inthe lexicon. Another method used to process natural language is toexamine parts of the input phrase for a standard pattern. For example, anumber including a colon and followed by "am" or "pm" fits the patternof a phrase indicating time. A different method used to process naturallanguage is to query a database that stores a large number of phrasesand match a phrase to one or more items stored in the database.

The abovementioned processes have typically been used with a fixed setof rules to interpret natural language. An a priori specification ofknown linguistic structures and their meaning is used to performsemantic interpretation. Each of the abovementioned processes haslimitations in versatility and the time required to match a phrase.Natural language processing of the prior art has thus been inefficientand rigid in its ability to recognize the meaning of natural languageutterances.

What is needed is a natural language processor that will quickly andaccurately identify and map natural language input phrases. What isfurther needed is a natural language processor that identifies phrasesaccording to a dynamic set of rules and data.

SUMMARY OF THE INVENTION

The present invention provides a natural language processor for the userof a computer system. The method of the present invention analyzesportions or substrings of a natural language input string to determinethe meaning of the input string. Large substrings are analyzed andreduced in size until the substrings can be mapped into a database usingmapping routines. Three dynamic mapping routines are preferably used insequence in the present invention: a phrasal processor, a patternprocessor, and a database query processor. These processors provide arobust method of identifying the meaning of natural language input.

More particularly, a method for deducing meaning from a natural languageinput comprises the steps of (a) receiving an ordered string of wordobjects having length and a natural language meaning, (b) selecting aword window length that is no greater than the length of the orderedstring, and (c) successively moving the word window along the orderedstring and analyzing the meaning of a substring of word objects thatfall within the word window. The substring is removed from the orderedstring if the substring has a recognized meaning, until all substringsof the ordered string that fit within the window have been analyzed. Ina step (d), the word window length is reduced and step (c) is repeateduntil only an unrecognized residual of the ordered string remains. Theordered string is preferably received from strokes entered by a styluson a display screen of a pen-based computer or from a microphonereceiving audible speech input. The step of analyzing the meaning of thesubstring preferably includes mapping the substring against a databaseusing one or more mapping routines. The mapping routines are preferablyarranged in a hierarchy, wherein a successive mapping routine is used toanalyze the substring when a previous mapping routine in the hierarchycannot map the substring. The method preferably further includes a stepof determining a computer-implemented task specified by the orderedstring using the recognized substrings and performing thecomputer-implemented task.

A method for recognizing meanings of natural language phrases comprisesthe steps of attempting to map at least a portion of an ordered stringof word objects into a database according to a phrasal mapping routine.If the string portion does not map into the database using the phrasalmapping routine, a step of attempting to map the portion into a databaseaccording to a pattern mapping routine is accomplished. If the patternmapping routine is unable to map the string portion, then a databasequery mapping routine attempts to map the string portion. The mappingroutines are preferably dynamic mapping routines referencing dynamicdata structures. Remaining portions of the ordered string are preferablymapped according to the mapping routines until all word objects of thestring have been analyzed.

A preferred apparatus in accordance with the present inventionimplements the natural language processor on a computer system, and morepreferably on a small, handheld computer system such as a personaldigital assistant (PDA). The natural language can be input into the PDAby a variety of methods including a tablet-and-stylus, an inputmicrophone, a keypad, etc.

Because the natural language processor first analyzes larger substringsof an input string and then reduces the size of the substrings, naturallanguage input is more quickly and reliably processed and its meaningdetermined. In addition, the present invention uses three dynamicnatural language processors in succession to allow greatly increasedsensitivity to the context of natural language input and enhancedcapability in interpreting the input.

These and other advantages of the present invention will become apparentto those skilled in the art upon a reading of the followingspecification of the invention and a study of the several figures of thedrawing.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a pen-based computer system in accordancewith the present invention;

FIG. 2 is a top plan view of the pen-based computer system of FIG. 1;

FIG. 3 Is a table illustrating the method of examining substrings of thepresent invention:

FIG. 4 is a flow diagram illustrating the method of processing naturallanguage of the present invention;

FIG. 5 is a flow diagram illustrating the step of developing a string ofFIG. 4;

FIG. 6 is a table illustrating a hierarchy of mapping routines used inthe present invention;

FIG. 7A is a flow diagram illustrating the step of obtaining a substringof FIG. 4;

FIG. 7B is a table showing the contents of the matrix HITARRAY;

FIG. 8 is a flow diagram illustrating the step of removing a substringand storing a mapping of FIG. 4;

FIG. 9 is a phrase look-up table used for the phrasal processor of thepresent invention;

FIG. 10 is a pattern look-up table used for the pattern processor of thepresent invention;

FIG. 11 is a type frame illustrating a database query processor of thepresent invention;

FIG. 12 is a task template used for determining a task in the presentinvention; and

FIG. 13 is a flow diagram illustrating the determine task and performtask steps of FIG. 4.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The present invention is well suited for pointer-based computer systemssuch as the pen-based, pen-aware and mouse controlled systems that arecurrently popular. The present invention is also well suited forcomputer systems implementing voice input and speech recognition. Forthe purposes of illustration, the invention will be described inconnection with a pen-based system incorporating voice input. However,the present invention is also suitable for other types of computersystems, such as mainframe systems, keyboard based systems, etc.

As shown in FIG. 1, a pen-based computer system 10 in accordance withthe present invention includes a central processing unit (CPU) 12, readonly memory (ROM) 14, random access memory (RAM) 16, input/output (I/O)circuitry 18, and a display assembly 20. The pen-based computer system10 may also optionally include a mass storage unit 22, a microphoneamplifier 23, a microphone 25, a keypad (or keyboard) 24, a serial port26, an infrared (I/R) port 28, a speaker amplifier 27, a speaker 29, anda clock 30.

The CPU 12 is preferably a commercially available, single chipmicroprocessor. While CPU 12 can be a complex instruction set computer(CISC) chip, it is preferable that CPU 12 be one of the commerciallyavailable, reduced instruction set computer (RISC) chips which are knownto be of generally higher performance than CISC chips. CPU 12 is coupledto ROM 14 by a unidirectional data bus 32. ROM 14 preferably containsthe basic operating system for the pen-based computer system 10. CPU 12is connected to RAM 16 by a bi-directional data bus 34 to permit the useof RAM 16 as scratch pad memory. ROM 14 and RAM 16 are also coupled toCPU 12 by appropriate control and address busses, as is well known tothose skilled in the art. CPU 12 is coupled to the I/O circuitry 18 bybi-directional data bus 36 to permit data transfers with peripheraldevices.

I/O circuitry 18 preferably includes a number of latches, registers anddirect memory access (DMA) controllers. The purpose of I/O circuitry 18is to provide an interface between CPU 12 and such peripheral devices asdisplay assembly 20, mass storage 22, amplifier 23, keypad 24, serialport 26, I/R port 28, and amplifier 27.

Display assembly 20 of pen-based computer system 10 is both an input andan output device. Accordingly, it is coupled to I/O circuitry 18 by abi-directional data bus 38. When operating as an output device, thedisplay assembly 20 receives data from I/O circuitry 18 via bus 38 anddisplays that data on a suitable screen. The screen for display assembly20 is preferably a liquid crystal display (LCD) of the type commerciallyavailable from a variety of vendors. The input device of displayassembly 20 is preferably a thin, clear membrane which covers the LCDdisplay and which is sensitive to the position of a stylus 38 on itssurface. With such a structure, the membrane of the display assembly 20can serve as an input "tablet." These position sensitive membranes arealso readily available on the commercial market. Alternatively, othertypes of tablets can be used, such as inductively coupled tablets.Combination display assemblies such as display assembly 20 which includeboth the LCD and the input membrane are commercially available from suchvendors as Scriptel Corporation of Columbus, Ohio.

Some type of mass storage 22 is generally considered desirable. Massstorage 22 can be coupled to I/O circuitry 18 by a bi-directional databus 40. However, the mass storage 22 can be eliminated by providing asufficient amount of RAM 16 to store user application programs and data.In that case, the RAM 16 can be provided with a backup battery toprevent the loss of data even when the pen-based computer system 10 isturned off. However, it is generally desirable to have some type of longterm mass storage 22 such as a commercially available miniature harddisk drive, nonvolatile memory such as flash memory, battery backed RAM,a PCMCIA card, or the like.

Microphone amplifier 23 is coupled to I/O circuitry by bus 41 andincludes well-known components which convert a speech signal input inmicrophone 25 to an analog signal, such as amplifiers, preamplifiers,etc. Amplifier block 23 can also include components which convert theanalog signal to digital signals that can be used with CPU 12, such asanalog to digital converters (ADC's). The recognition of words spoken bya user can be implemented in software executed by CPU 12, and is wellknown to those skilled in the art.

The keypad 24 can comprise an array of mechanical buttons or switchescoupled to I/O circuitry 18 by a data bus 42. Alternatively, keypad 24can comprise an entire, standard QWERTY keyboard. In the presentembodiment, a separate keypad 24 is not used in favor of a "pseudo"keypad 24'. This "pseudo" keypad 24' comprises "button" areas which areassociated with a bottom edge of the tablet membrane that extends beyondthe lower edge of the LCD display. These button areas are defined by aprinted or silk-screened icons which can be seen through the transparentmembrane of the input tablet. When the "buttons" are selected byengaging the stylus 38 with the membrane over these printed icons, themembrane senses the pressure and communicates that fact to the CPU 12via data bus 38 and I/O 18. An example of pseudo keypad 24' is shown inFIG. 2.

Serial port 26 is coupled to I/O circuitry by a bi-directional bus 44.The serial port 26 can be used to couple the CPU to external devices andnetworks.

Infrared (I/R) port 28 is coupled to I/O circuitry by a bi-directionalbus 46. The I/R port can be used for outgoing information (e.g. tocontrol a printer or some other external device, or to communicate withother computer systems) or for incoming information from other computersor devices.

Speaker amplifier 27 is coupled to I/O circuitry by a bus 47. Amplifier27 is used to drive a speaker 29 with signals output by CPU 12 and canalso include components such as filters, specialized sound chips, etc.Speaker 29 can be used for audio output such as sound effects,synthesized voice, and other user feedback.

Clock 30 preferably comprises a real-time clock to provide real-timeinformation to the system 10. Alternatively, clock 30 can simply provideregular clock pulses to, for example, an interrupt port of the CPU 12which can count the clock pulses to provide the time function. However,this alternative clock embodiment tends to be wasteful of CPU processingpower. Clock 30 is coupled to CPU 12 by a data bus 48.

In operation, information is input into the pen-based computer system 10by "writing" on the screen of display assembly 20 with the stylus 38.Information concerning the location of the stylus 38 on the screen ofthe display assembly 20 is input into the CPU 12 via data bus 38 and I/Ocircuitry 18. Typically, this information comprises the Cartesian (i.e.x & y) coordinates of a pixel of the screen of display assembly 20 overwhich the tip of the stylus is positioned. Commercially availablecombination display assemblies such as the aforementioned assembliesavailable from Scriptel Corporation, Sharp, and others includeappropriate circuitry to provide the stylus location information asdigitally encoded data to the I/O circuitry of the present invention.The CPU 12 then processes the data under control of an operating systemand possibly an application program stored in ROM 14, RAM 16, or massstorage 22. The CPU 12 next produces data which is transferred to thedisplay assembly 20 via I/O circuitry 18 and data bus 38 to produceappropriate images on the screen portion of the display assembly 20.

In FIG. 2, the pen-based computer system 10 of FIG. 1 is shown housedwithin a generally rectangular enclosure 50. The CPU 12, ROM 14, RAM 16,I/O circuitry 18, and clock 26 are preferably fully enclosed within theenclosure 50. The display assembly 20 (FIG. 1) is mostly enclosed withinthe enclosure 50, but a viewing screen 52 of the display assembly isexposed to the user. As used herein, the term "screen" will refer to theportion of the display assembly 20 which can display an image that canbe viewed by a user. Also accessible to the user is the pseudo keypad24' that was described with reference to FIG. 1.

Upon power-up, pen based computer system 10 displays on screen 52 aninitial "note" area 54a including a header bar 56a and a number ofguidelines 58. The header bar 56a preferably includes the date ofcreation of the note area 54a and a number of icons and "soft" buttons,not particularly germane to the discussion of the present invention. Forthis reason, the header bar 56a will not be discussed in detail herein.The optional guidelines 58 aid a user in entering text, graphics, anddata into the pen-based computer system 10.

Additional note areas, such as a note area 54b, can be formed by theuser by drawing a substantially horizontal line across the screen 52with the stylus 38. The substantially horizontal line is recognized bythe system 10 and is converted into a second header bar 56b. Additionaltext, graphical, and other data can then be entered into this secondnote area 54b. For example, the text object T is a phrase that has beenentered into second note area 54b. In the described embodiment, textobject T was originally entered as several strokes of "ink" which wererecognized by the CPU and converted into printed words of the object T.Text object T can also be entered by speaking words into microphone 23which are recognized and displayed.

In this preferred embodiment, the keypad 24', as explained previously,comprises a printed or silk-screened member 60 provided beneath a loweredge of a thin, clear, stylus-sensitive membrane 62 of the input"tablet." Alternatively, a keypad could comprise a mechanical keypad (orkeyboard) 24, or a keypad could comprise "soft buttons" i.e. imagesgenerated at convenient locations on the screen 52, in which case a"button" would be activated by touching the stylus to the screen overthe image of the button. The keypad 24' preferably includes a number ofdedicated function buttons 64 and a pair of scroll buttons 66a and 66b.The operation of the note areas 54a, 54b, etc., scroll buttons 66a and66b, and other aspects of computer system 10 are discussed in greaterdetail in co-pending U.S. patent application Ser. No. 07/868,013, filedApr. 13, 1992 on behalf of Tchao et al., assigned to the assignee of thepresent invention and incorporated herein by reference in its entirety.

The screen illustrated in FIG. 2 is referred to as the "notepad", and ispreferably an application program running under the operating system ofthe pen based computer system 10. In this preferred embodiment, thenotepad is a special or "base" application which is always availablebeneath higher level applications. The notepad application, like otherapplications, run within a window, which in this instance comprises theentire viewing screen 52. Therefore, as used herein, a "window" is theentire screen or any portion of an entire screen which is dedicated to aparticular application program.

A status bar 68 is provided at the bottom of the notepad application.The status bar 68 is provided with a number of active and display areas,which again are not particularly germane to the present invention andwill therefore not be discussed in detail herein. U.S. patentapplication ser. Ser. No. 07/976,970 filed Nov. 16, 1992 on behalf ofFoster et. al, entitled "Status Bar for Application Windows" andassigned to the assignee of the present invention describes how to makeand use the status bar, and is incorporated herein by reference in itsentirety.

The enclosure 50 is preferably provided with apertures 70 which permitthe free transmission of sound from speaker 29 which is housed withinenclosure 50. The speaker can be used to provide user feedback, or totransmit audible information to a user. Enclosure 50 is also preferablyprovided with apertures 71 which permit the reception of sound from anexternal source to microphone 25. A user can preferably speak intoapertures 71 to provide voice input to the microphone speaker 23.Alternatively, apertures 70 and speaker 29 can be used as a dualfunction speaker/microphone.

In the present invention, objects are preferably implemented as part ofa frame system that comprises frame objects related by a semanticnetwork. A description of semantic networks can be found in "AFundamental Tradeoff in Knowledge Representation and Reasoning",Readings in Knowledge Representation, by Brachman and Leveseque, MorganKaufman, San Mateo, 1985.

It will be noted there is a liberal use of graphic elements in thepresent invention. For example, the header bars 56a and 56b includelines and other graphical elements. Processes for drawing lines on acomputer screen are well known to those skilled in the art. For example,graphics software such as QUICKDRAW from Apple Computer, Inc. ofCupertino, Calif. can be used to draw lines, simple geometrical shapes,etc. A description of the QUICKDRAW graphics software is found in thebook Inside Macintosh, Volumes I, II, and III, by C. Rose et al.,Addison-Wesley Publishing Company, Inc., July 1988. With such graphicssoftware, a line can be drawn by simply specifying the coordinates ofthe beginning and the end of the line, and by specifying the width ofthe line.

Another preferred tool for implementing the system of the presentinvention is a view system. Various types of view systems are well knownto those skilled in the art. In the present system, the notepadapplication on the screen 52 forms a first or "root" layer, with thestatus bar 68, for example, positioned in a second layer "over" the rootlayer. The various buttons of the status bar 68 are positioned in athird layer "over" the second and root layers. The view systemautomatically handles "taps" and other gestures of the stylus 38 on thescreen 52 by returning information concerning the tap or gesture and anyobject to which it may be related. Again, the status bar 68 and the viewsystem is described in greater detail in copending U.S. patentapplication Ser. No. 07/976,970, which has been incorporated herein byreference. It is therefore clear that the object oriented programmingand view system software makes the implementation of the processes ofthe present invention less cumbersome than traditional programmingtechniques. However, the processes of the present invention can also beimplemented in alternative fashions, as will be well appreciated bythose skilled in the art.

A method for processing natural language of the present invention is aprocess that is continuously running on the system 10, i.e. the methodmonitors input and events continuously. The method can be, considered tobe an extension of the operating system which continuously notices userbehavior.

The method of the present invention is well suited for use withcomputerized assistance. For example, a method for deducing user intentand providing computer implemented services is disclosed in co-pendingparent U.S. patent applications Ser. No. 08/099,861 and Ser. No.07/889,225, which were previously incorporated by reference.

FIG. 3 is a table 72 illustrating the method used in the presentinvention for mapping substrings (portions) of an ordered string into adatabase to recognize the substrings and determine acomputer-implemented task specified by the ordered string. The column onthe left lists several substrings 73, which are a number of consecutiveword objects of an ordered input string that has been delineated fromuser input. In the example of FIG. 3, text object T as shown in FIG. 2is used as the ordered input string, which includes the word objects"Lunch with Bill Monday 12:30 pm at Chez Sovan." Each entry in the leftcolumn shows a different substring being displayed by a "word window",which herein is a window having a specific word object length which canbe moved along the ordered string to "display" (select) different wordobjects of the ordered string which fall within the boundaries of thewindow. The meaning of the substring displayed by the word window isanalyzed, i.e. the substring is examined and attempted to be mappedagainst a database using a mapping routine. The right column of table 72lists the length and position of the word window for each substringlisted in the left column of the table. LENGTH specifies the length ofthe word window, and therefore of the displayed substring, in amount ofrecognized word objects. POS specifies the position of the left edge ofthe word window in relation to the entire input string, where positionnumber 1 is designated to be the first word, or the left end, of theordered string. For example, the third substring listed in table 72shows a word window having a length of 8, meaning the word window isdisplaying a substring including 8 word objects. The position POS is 2,meaning that the left edge of the word window is located at word objectnumber 2 of the ordered input string.

The successive substrings listed in table 72 show the method and orderof analyzing the ordered string according to the present invention inorder to map substrings against a database and determine a taskspecified by the input string. In the described embodiment, the lengthof the word window is initially set to the largest length of the orderedstring so that the largest possible substring (shown at the top of table72) is attempted to be mapped first. In other embodiments, the intialword window length can be set smaller than the full length of theordered string by a predetermined number of word objects.

The process then attempts to map the substring displayed by the wordwindow into a database using mapping routines, i.e. the process attemptsto recognize the meaning of the substring displayed by the word windowso that the ordered string can be implemented as a task. Each substringis examined by a successive mapping routine in a predetermined hierarchyavailable to the computer system, where a successive mapping routine inthe hierarchy is used if the substring does not map using the previousmapping routine. This process is described with reference to FIG. 4.

The displayed (currently selected) substring either maps or does not mapinto the available mapping database(s). If the substring can be mapped,the substring is removed from the ordered string. The word window isthen moved along the ordered string to display the next substring to theright of the mapped substring. The new substring is then analyzed andattempted to be mapped into a database. If the displayed substringcannot be mapped, the word window is moved one word object to the rightand the next substring is analyzed. In the described embodiment, theword window moves from left to right; however, the word window can alsobe moved in other directions, such as right to left. In otherembodiments, the word window can be moved more than one word objectalong the ordered string if a substring cannot be mapped.

Once the word window is moved to the right end of the ordered string,all the substrings having the length of the word window have beenanalyzed. The word window is then reduced in length and is positioned atthe left end of the ordered string. The word window is then moved alongthe ordered string in a similar fashion to that described above. Thelength of the word window is thus successively reduced until the lengthof the word window is zero. At that time, all of the word objects of theordered string have been analyzed and are either mapped into a databaseor cannot be mapped using the available mapping routines.

In the example of FIG. 3, the first substring analyzed in table 72 isdisplayed in a word window having a length of 9 word objects (which isthe total length of the input string) and a position of 1. Thisdisplayed substring fails to map into the available databases. Since theword window is already at the right end of the ordered string, the wordwindow cannot be moved along the string any further. The length of theword window (and the substring) is thus reduced to 8 and the position ofthe word window is set to position 1 as shown in the second row of theleft column. The substring at position 1 is analyzed, and the wordwindow is moved to position 2. The displayed substring at position 2also fails to be mapped. The word window's length is reduced to 7 and itis positioned at position 1 at the fourth row of table 72, and theprocess continues in a similar fashion.

Eventually the size of the substring becomes small enough so that asubstring can usually be mapped using a mapping routine. Substring 74,"Monday 12:30 pro", for example, has been mapped into a database using apattern matching mapping routine when the word window has a length of 3and a position of 4. The brackets surrounding substring 74 in FIG. 3indicate that the substring has been mapped. Substring 74 is stored andis removed from the input string, and the process continues with theremaining word objects of the input string. Substring 76, "Lunch with"has been mapped using the phrasal matching mapping routine. Substrings78 have been mapped using a database query mapping routine. The finalword remaining is substring 80, "at", which cannot be mapped using anyof the mapping routines and thus is a "residual" word from the process.The computer system then determines and implements a task according tothe recognized (mapped) substrings, as detailed below with reference toFIG. 4.

A method or process 80 for processing natural language in accordancewith the present invention is shown in FIG. 4. The process begins atstep 82 wherein a user is interfacing with the computer system. In astep 84, an input string of recognized words is developed by the CPU 12from natural language input received from the user. The process ofdeveloping an input string is described in more detail with reference toFIG. 5.

In next step 84, the CPU arranges NMAP mapping routines in a hierarchy.As described above, the mapping routines are methods of mapping theordered input string to specific objects in a database that arecomponents of a task that is to be implemented by the computer system.Step 86 is described in greater detail with reference to FIG. 5. Theprocess then continues to step 88, wherein a variable LENGTH is setequal to NWORDS, and a variable POS is set equal to 1. Variable LENGTHstores the length, in word objects, of the word window, which is alsothe length of the substring being analyzed, and variable NWORDS is equalto the total number of word objects in the ordered string. Variable POSstores a number that represents the word position of the word window,which is the position that the examined substring begins, i.e. theoffset from the beginning of the input string. In next step 90-94, anarray of values called HITARRAY is initialized to 0. In step 90, acounter i is set to zero and the CPU checks if i is less than or equalto NWORDS. If it is, then step 92 is implemented, in which countervariable j is set to 1 and the CPU checks if j is less than NWORDS -i+1.If it is, then step 94 is implemented, in which HITARRAY(i,j) is set to0. The nested loops continue until i is greater than NWORDS in step 90,at which point HITARRAY is completely initialized and the processcontinues to step 96. HITARRAY is described in further detail below withreference to FIG. 7a.

In step 96, the CPU obtains a substring from the input string. This isaccomplished by moving the word window over a number of word objects inthe input string and is described in more detail with reference to FIG.7. After a substring has been obtained, step 98 in implemented, in whichthe CPU checks if the input string has been fully analyzed, i.e. if allword objects of the input string have been mapped or are not able to bemapped by the CPU. This is preferably accomplished by examining thevariable LENGTH; if LENGTH=0 or null, then all of the word objects ofthe input string have been analyzed. If the input string has not beenfully analyzed, then step 99 in implemented, in which the CPU marks a"hit" in HITARRAY; in the described embodiment, this is accomplished bysetting the value at HITARRAY (LENGTH,POS) to 1. Step 100 is thenimplemented, in which a counter variable j is set to 1 and checked if itis less than the variable NMAP, which is the number of mapping routinesbeing used by the computer system. While j is less than or equal toNMAP, step 102 is implemented, in which the CPU checks if the substringmaps into a database using mapping routine(j). If the substring does notmap using mapping routine(j), the process loops back to step 100, inwhich j is incremented and checked if it is less than or equal to NMAP,and step 102 is again implemented for the next mapping routine in thehierarchy. The obtained substring is thus analyzed using all availablemapping routines in the order of the mapping routine hierarchy until itis either mapped into a mapping routine or not mapped into any of themapping routines. In the described embodiment, the mapping routines withless computational cost are arranged first in the hierachy for moreefficiency. Thus, if a substring can be mapped using a less complex andtime consuming mapping routine, the other more complex mapping routinesdo not have to be implemented for that substring.

There are several well-known mapping routines known to those skilled inthe art. As described below with respect to FIG. 6, three mappingroutines are described herein: a phrasal processor, a pattern processor,and a database query processor. These mapping routines match (map) asubstring against one or more objects (entries) in a "knowledge base".As used herein, the term "knowledge base" refers to a collection ofspecialized databases storing objects of a particular category which canbe matched to substrings. For example, using a phrasal processor, theCPU accesses the knowledge base by looking for a match with thesubstring in a phrasal lexicon or look-up table, which is typicallystored in memory or on a storage device such as a hard disk drive. Aphrasal look-up table is described in more detail with reference to FIG.9. Using a pattern processor, the CPU accesses the knowledge base bylooking for a pattern stored within a hierarchy on the knowledge basewhich corresponds to a pattern within the substring. This process isdescribed in more detail with reference to FIG. 10. For a database queryprocessor, the CPU sends the substring as a query to a database andmatches related entries in the database to the substring. This processis described in more detail with reference to FIG. 11. The knowledgebase includes the hierarchical information used to assign categories toa phrases so that a task can be assigned to an input string, asdescribed with reference to FIG. 12.

If the substring maps into a database using mapping routine(j) at step102, step 104 is implemented, in which the substring is removed from themapping process and the mapping is stored for later use. This step isdescribed in more detail with reference to FIG. 8. The process thenloops back to step 96, in which the word window is moved and a differentsubstring is obtained from the input string.

If no mapping for the substring is found using any of the mappingroutines in the hierarchy, the process loops back to step 96 to obtain anew substring which is either smaller than the substring which could notbe mapped or is a different portion of the input string (detailed withrespect to FIG. 7).

The process loops until all of the word objects of the input string arefully analyzed, after which the process flows from step 98 to step 106.In step 106, the "residual" from the input string is stored. Theresidual is the remaining words of the input string which could not bemapped from any of the mapping routines being used. Residuals are oftenconnecting words such as "in" or "at" which have no inherent meaning forperforming a task, as shown in the example of FIG. 3. In step 108, theCPU determines the task which is based upon the mappings performedearlier in the process. This step is described in more detail withreference to FIGS. 12 and 13. Once the task has been determined by theCPU, then step 110 is implemented, in which the recognized task isperformed by the CPU. The process is then complete as indicated in step112.

Although the present invention is described in terms of performing acomputer-implemented task such as faxing, printing, etc. (see FIG. 12),other processes are also suitable for use with the method describedherein. For example, the method of the present invention can be used todetermine the meaning of an inputted phrase and output an interpretationof the inputted phrase using different terminology. Thus, a specificphrase could be "paraphrased" in simpler (or more complex) language,e.g. for educational purposes.

FIG. 5 is a flow diagram illustrating step 84 of FIG. 4, in which astring is developed from user input in the form of natural language. Theprocess begins at step 116, and, in step 118, the CPU receives userinput, preferably as natural language in spoken or written form. Naturallanguage is characterized by phrases which the user enters in a naturalform, i.e., in a form which is not constrained by rules which thecomputer must follow to process input. For example, a natural languagephrase meaning "I want to schedule a lunch meeting with Bill on Mondayat 12:30 pm at Chez Sovan Restaurant" can be entered in several forms.One way could be "Lunch with Bill Monday 12:30 pm at Chez Sovan," asshown above. A different form could be "Have lunch 12:30 pm Monday atChez Sovan with Bill."

One form of input suitable for the present invention is pen or stylusinput, in which the u "inks" input on a screen such as display 20 with astylus. Another form of user input is voice input, in which the user,for example, speaks into microphone 25 and the speech is converted intodigital signals to be analyzed by the CPU. A different form of userinput is keyboard input, wherein a user types in data or selects keys orbuttons to input data. Other forms of input are also suitable for thepresent invention. A user can select a portion of text that is alreadybeing displayed by the computer system; the selected or highlighted textcan be considered the input which the user wishes to be processed by themethod of the present invention. User input can also be received fromother input devices, such as storage devices, pointing devices, andcommunication devices (i.e. modems and the like).

In step 120, the user input is recognized. That is, each word object ofthe user input is recognized as a word in English or whatever selectedlanguage is being used. Herein, a "word object" is an object in the userinput which is recognized, such as words or numbers. Thus, if penstrokes are entered as user input is step 118, the strokes arerecognized in step 120 and divided into separate word objects, which canbe made up of individual characters or letters. If the user input isvoice input, the speech is recognized and similarly converted into wordobjects. Word objects in written natural language inputted by the userare preferably delimited by spaces or null characters, and word objectsin spoken natural language are preferably delimited by pauses of apredetermined length of time. Software used for recognition of varioustypes of user input is well known to those skilled in the art. Themethod of the present invention preferably closely cooperates with therecognition architecture of the operating system. The recognitionarchitecture preferably recognizes higher-level meanings from strokesmade by the stylus 38 on the screen 42 of display assembly 20 and fromvoice input spoken into microphone 25. While the method of the presentinvention is not limited to any one recognition architecture, the penstroke recognition of the present invention is particularly well adaptedto operate with the recognition architecture described in co-pendingU.S. patent applications Ser. No. 07/889,216, filed May 27,1992 onbehalf of Beenlnk et al., entitled "Recognition Architecture andInterface", and U.S. patent application Ser. No. 08/001,123, filed Jan.5, 1993 on behalf of G. Pagallo, entitled, "Method and Apparatus forComputerized Recognition," both assigned in common with the presentapplication to Apple Computer, Inc., the disclosures of which are herebyincorporated by reference in their entirety. The speech recognition ofthe present invention can be used with speech recognition products soldon the market, such as PlainTalk by Apple Computer, Inc.

Step 122 is next implemented, in which a input string (i.e. phrase) isdelineated from the recognized user input. The length of the string isdetermined by the amount of input data; the CPU preferably marks the endof an input string by checking for a timeout or a spaceout. A timeoutoccurs when a predetermined length of time has expired after the lastword is input into the computer system; if no additional words are inputwithin that length of time, the CPU assumes that the user is finishedentering input and delineates a string from the input. A spaceout occurswhen the user writes with a stylus or other pointing device on an areaof display screen 20 that is a predetermined distance from thepreviously-written words and letters of the input data. If a spaceoutoccurs, the CPU similarly assumes that the user has finished enteringdata and delineates a string from the inputted data. The delineatedstring is preferably made up of ordered word objects which are delimitedby a space or null located between each word object. With a stringdelineated from the user input, the process is complete as indicated instep 124.

FIG. 6 is a chart 126 illustrating step 86 of FIG. 3, in which NMAPmapping routines are arranged in a hierarchy. In FIG. 6, the hierarchylevel 128 is shown as the left column, the specific mapping routine 130is shown in the middle column, and the computational cost 132 is shownin the right column. In the described embodiment, a phrasal processor isdesignated the first mapping routine in the hierarchy, followed by apattern processor as the second mapping routine and database queryprocessor as the third mapping routine. Other types of mapping routinescan also be used; for example, more specialized mapping routines can beused for applications of specific subjects. Also, mapping routines canbe added to or removed from the hierarchy. In the preferred embodiment,the mapping routine of level 1 is the mapping routine having the lowestcomputational cost. This means that the level 1 mapping routine examinesand processes the input string in the least amount of time and using theleast amount of computational cycles. Conversely, the last level mappingroutine (level 3 in the described embodiment) preferably is the mostcomplex mapping routine which has the highest computational cost andtakes the most amount of computational cycles to process substrings ofthe input string. The higher level mapping routines are used only whenthe lower level routines are unable to map a substring.

FIG. 7A is a flow diagram illustrating step 96 of FIG. 3, in which theCPU obtains a substring from the input string. The process begins at astep 136, and, in a step, 138, the CPU checks if the variable LENGTH,representing the length of the word window, is equal to zero. If so,step 140 is implemented, in which LENGTH is set equal to null,indicating that all of the word objects of the input string have beenanalyzed by the mapping routines. The process is then complete asindicated at step 142.

If LENGTH does not equal zero at step 138, then step 144 is implemented.In step 144, the CPU checks the HITARRAY matrix for the value stored inthe array at location (LENGTH, POS). The first time through the process,LENGTH is equal to NWORDS and POS is equal to 1 from the initializationin step 88 of FIG. 4. In step 146, the CPU examines the value obtainedfrom HITARRAY and determines if the substring is "possible" or not.Herein, a substring that is "possible" means that the substring has notyet been analyzed, i.e. it has not yet been displayed in the wordwindow. A substring that is not possible means that the substring hasalready been analyzed by the mapping routines.

A table 147 showing the contents of the matrix HITARRAY is shown in FIG.7B. The first column 148 shows the values that POS (position) maycontain. The second column 150 shows the values that LENGTH may contain.In the example of table 147, NWORDS is initially equal to 9, the ninewords of the input string shown as text object T in FIG. 2. WhenLENGTH=9, the length of the examined substring is 9 words, which is theentire string. There is thus only one position for the length 9substring: position i. If this examined substring cannot be mapped intoa mapping routines, LENGTH is dermenmd to a lesser value, and morpositions of the substring can be assigned values in the table. Forexample, if LENGTH=3, then there are seven different positions that asubstring of length 3 can possess within the 9-word string. Each ofthose seven positions is listed in table 147.

For each entry in table 147, a value of HITARRAY(LENGTH, POS) is listedin column 152. As shown in FIG. 7a, the VALUE column is initialized toall zeros in steps 90-94 of FIG. 4. Thus, all the positions of allpossible substrings are set to "possible". Once a substring is analyzedby the mapping routines, its value in table 147 is set to "1" as shownin step 99 of FIG. 4. The maximum number of value entries for allpossible lengths and positions of substrings is equal to (NWORDS²+NWORDS)/2.

Referring back to FIG. 7, in step 146 the CPU checks HITARRAY for thecurrent values of LENGTH and POS. In the preferred embodiment, LENGTH isinitially set to the maximum number of words and is decremented if amapping cannot be found for the larger substring. It has been found thatmore phrases are mapped accurately and a task is executed more inaccordance with the user's intent if larger substrings are examinedbefore reducing the size of the substring. Mapping a substring to amapping routine is described in more detail below with reference to FIG.8. If the CPU finds that the current substring is possible, then step154 in implemented, in which the substring is set equal to the stringdefined by the current values of POS and LENGTH. For example, ifLENGTH=9 and POS=1, then the substring is set equal to the entire inputstring T as shown in FIG. 2, which has 9 words. If LENGTH=5 and POS=3,then the substring is set equal to 5 consecutive words of the inputstring starting from the third word position; using text object T, thesubstring would thus be equal to "Bill Monday 12:30 pm at." Once thesubstring is thus obtained, the process is complete as indicated in step142.

If the CPU determines in step 146 that the current POS and LENGTH do nothave a "possible" (0) value, then step 156 is implemented. In step 156,POS is incremented. In next step 158, the CPU checks if POS+LENGTH isgreater than NWORDS+1. If it is not, the process loops back to step 138to check if LENGTH is equal to zero. If the sum of step 158 is greaterthan NWORDS+1, then step 160 is implemented, in which POS is set equalto 1 and LENGTH is decremented. Once POS+LENGTH is greater thanNWORDS+1, then that length of substring has been examined at allpossible positions of the input string. The length of the word window isthen decremented to see if a smaller substring can be mapped into amapping routine. Once step 160 has been implemented, the process thenloops back to step 138.

FIG. 8 is a flow diagram illustrating a process used in step 104 of FIG.4, in which a substring has been recognized and wherein the CPU stores amapped substring and removes that substring from the input string. Theprocess begins at step 162. In step 164, the PU stores the the mappingof the substring (i.e. the object or type category in the knowledge basematched to the substring) and removes the substring from the inputstring. The word objects of the substring are removed from the inputstring so that they will not be examined in further loops of theprocess. In the described embodiment, the word objects are flagged sothat the CPU will ignore them. In step 166, NWORDS is updated to the newtotal number of word objects in the input string after the mappedsubstring has been removed by subtracting the number of word objects inthe removed substring from NWORDS. In step 168, HITARRAY(LENGTH,POS) ismarked as "possible", i.e., in the described embodiment, the value ofHITARRAY(LENGTH,POS) is set to zero. This step is implemented becausethe substring at the current POS value has been removed, and any wordobjects in the input string which are positioned after the removedsubstring are now at position POS. These word objects have not beenanalyzed by the mapping routines, and thus should have a "possible"(0)status in HITARRAY. Once step 168 is complete, the process is completeas indicated by step 170.

FIG. 9 illustrates a phrase look-up table 172 used to match a substringto entries in the knowledge base. Table 172 includes a left side 174 ofthe table showing a list of substrings (phrases) and a right side 176 ofthe table showing a list of frame instances, i.e. occurrences within theknowledge base, of each of those phrases. For example, it can be seenthat the phrase ISAAC has the three instance frames <PERSON-1>,<PERSON-2>, and <PERSON-3>. The phrase "Lunch with" has the one instanceframe <MEET-1>. Similarly, the phrase "Schedule lunch with" has theinstance frame <MEET-2>. Table 172 is used as a "phrasal lexicon" ofoften-used phrases. The CPU can map a substring to an instance frameusing look-up table 172 in a small amount of time compared to the othermapping routines described herein.

A "frame" system is a preferred data structure used herein and is a formof semantic network. As is well known to those skilled in the art, asemantic network is a knowledge representation system where unarypredicates are treated like types and binary predicates are treated likeattributes. Further, the types are arranged in a taxonomy wheresubassumption holds. For example, for every item x, predicate2(x)implies predicate1(x) if predicate2 is a specialization of predicate1,i.e. DOG(FIDO) implies MAMMAL(FIDO).

A second aspect of a semantic network is the notion that there exists abinary relation called an attribute that maps unary predicates. Forexample, consider the unary predicates DOG(FIDO) and COLOR(BROWN). Anattribute may exist called COLOR-OF that can relate the unary predicatesas follows: ##EQU1##

A semantic network can be represented by a directed graph where thenodes are either types or constants and the edges are either labeledwith an attribute or with the special label IS-A. To find out, forexample, if a particular person in the knowledge base has a certainattribute, a search is commenced at a node representing that person andtraveling up the IS-A links until a node is found with an edge labeledwith the attribute. Therefore, graph traversal is the main source ofinference in a semantic network.

As noted above, semantic networks are well known to those skilled in theart of building knowledge bases. A description of semantic networks canbe found in "A Fundamental Tradeoff in Knowledge Representation andReasoning", Readings in Knowledge Representation, by Brachman andLeveseque, Morgan Kaufman, San Mateo, 1985.

The frame system is an elaboration of a semantic network. See, Brachmanand Leveseque, supra. Frame systems elaborate on the structure of typesthemselves and their attributes. Specifically, frame systems have threeaspects:

1) Values of attributes are stored explicitly or stored as a defaultvalue that the individual slot can inherit. This effectively caches someof the graph traversal.

2) Value or role restriction are constraints that must be satisfied byattribute values. These restrictions can constrain a value to be of acertain type (known as value class), of a certain maximum or minimumcardinality (in the case of multivalued slots), or a combination ofboth.

3) Attached procedures (also known as daemons or angels) that areinvoked when a value is accessed (either by getting or setting). Thisallows values to be computed on-the-fly. The procedures can becompletely arbitrary or be expressed in a restricted language. In eithercase, the procedure returns the computed value and whatever side-effectsthat may have occurred.

Frames used in the present invention have a number of slots which maycontain data, daemons, or other frames. Slots are accessed by makingassertions to the knowledge base. For example, if it was desired toretrieve all of the frames that were colored red, a typical frameaccessor language query would be in the form of:

    (QUERY (MEMBER-VALUE COLOR ?×RED)

and would return a list of frames that have a COLOR slot whose value isred. Compound queries can also be made.

In particular, FIG. 9 shows list 172 as a special case of a frame,referred to commonly as a "type" frame, since the frame refers to otherparticular types of frames, i.e., the types <PERSON>, <COUNTRY>, <MEET>,etc. Particular instances of the types are shown as instance frames.

Frame 178 is an instance frame of the type frame <MEET>. A slot in theinstance frame 178 holds the phrase, "lunch with." As depicted by thearrow 180, there is a "IS-A" link between the instance frame 187 and thetype frame 172 for traversal of the semantic network. When the phrase"lunch with" has been matched to the type frame <MEET>, a "semanticattribute tag" for the type frame <MEET> is stored as the mapping intothe knowledge base. A semantic attribute tag is a reference which pointsto a type frame in the knowledge base and informs the CPU where toaccess a piece of information in the knowledge base and what type ofinformation it is.

The information in the phrase look-up table is dynamic. That is, theinstance frames located in a type frame, and the type frames themselves,can be changed by a user. More frames can be added by the user to covera specific range of topics and/or different types of tasks. Since themethod of the present invention uses a knowledge base to referenceinformation, the CPU can refer to a dynamic body of data in look-uptable 174 to match a substring to a specific frame object. The presentinvention is thus not limited to a fixed set of phrases nor a fixeddatabase structure for accessing the phrases. The dynamic knowledge basestructure allows a flexible, customized system for mapping a substringto an object or a task.

FIG. 10 illustrates the use of a pattern processor to match a substringto an object in the knowledge base. A pattern table 184 has a column ofpatterns on its left-hand side and a column of instance frames on itsright-hand side. Two examples of patterns include "Monday 12:30 pm" and"12:30 Monday", both of which signify the time 12:30 pm. The patternsfor these two substrings are as follows: ##EQU2## as noted in table 184.Essentially, the first pattern for lunch time/date is a weekday string(i.e. "Monday", "Tuesday", Wednesday", etc.) followed by a number,followed by a colon, followed by another number, and finally followed by"pm". The second pattern is similarly constructed with the weekday atthe end of the pattern and no "pm" specified. In either case, a match tothe pattern look-up table would produce the instance frame <TIMESLOT>.

When a substring matches the first pattern <weekday> <num> ":" <num>"pm", an instance frame 186 <TIMESLOT> is made. Within frame 186 thereis at least one slot, which comprises another frame 188. Frame 188 isnamed <TIMESLOT-1>, and thus is the first instance of frame type<TIMESLOT>. The first slot within frame 188 includes the string "Monday12:30 pm" which matched into the pattern table. The second slot in frame188 includes the numeric value of the string. For example, Monday can beassigned the number 2, so that the number in the second slot of frame188 is 7:12:30:00. The pattern processor thus returns a semanticattribute tag <TIMESLOT> which is a mapping into the knowlege base for atime/date category substring. As with the phrasal processor (look-uptable) in FIG. 9, the CPU can refer to a dynamic database of patterns;the user can add or remove different patterns as desired. The patternprocessor thus does not have a specific, rigid set of patterns, butinstead has a changing set which maximizes context sensitivity to asubstring.

FIG. 11 illustrates accessing the knowledge base through a databasequery. Frame 192 is a type frame of the type <PERSON> which is stored inthe knowledge base. The data stored in the knowledge base can be storedin a separate database. A database particularly suitable for thedatabase query processor is known as a "soup", wherein data structuresused in the soup are not predetermined or fixed. Frames thus may havediffering numbers of slots, attributes, etc. Frame 192 has a number ofslots for various attributes applicable to a person, such as NAME,BIRTHDAY, TELEPHONE, FAX, etc. Frame 192 also includes a list of allframes which are an instance of the type frame <PERSON>, namely<PERSON-1>, <PERSON-2>, and <PERSON-3>.

Frames 194a, 194b, and 194c are instance frames of the type frame<PERSON>. For example, frame 194a is instance <PERSON-1>of frame 192 andhas the name <BILL-1>. Frame 194a is provided with the same slots astype frame 192, except some of the slots are filled with data. Forexample, the NAME slot of frame 194a is filled with the name "BillSetag." Unfilled slots have a null value or can be set to a defaultvalue. Likewise, frames 194b and 194c have slots filled with data abouttheir Bills. As depicted by the arrow 196, there is an IS-A link betweenthe instrances frames 194a-194c and the type frame 192 for traversal ofthe semantic network. Like the other mapping routines, the databasequery processor is dynamic in that a user can add or remove any numberof frames to the knowledge base to modify the structure and informationcontained therein. The database query is a much more general mappingroutine than, for example, the phrasal look-up table of FIG. 9. Thedatabase query can potentially search an entire database for a frameobject that matches the substring. A database query thus has a muchgreater computational cost than the phrasal processor and the patternprocessor, which use a limited lexicon of data. Since the examination ofseveral database frames can take a long time, the database queryprocessor is preferably used only if other, less costly mapping routinescannot map the substring.

When the substring "Bill" is queried to a database and found, thedatabase query processor recognizes through the knowledge base that"Bill" is a name and therefore information for this substring will befound in the <PERSON> type frame. As is well known to those skilled inthe art, a description of the knowledge base frames can be accessed tointerpret the database query. For example, the query "Bill" can be foundin the knowledge base description to refer to a <PERSON> type frame.Once the <PERSON> type frame is accessed, the instance frames<PERSON-1>, <PERSON-2>, and <PERSON-3> can be accessed. If informationabout a specific "Bill" is required, one of the "Bill" instance framesmust be chosen, since there are three Bill instance frames in thedatabase shown in the example of FIG. 11. The selection of the BILLinstance frame that most likely corresponds to the user's intendedperson can be based on several different criteria. One method is to lookat the number of instances that each BILL frame was selected by the userin the past. The BILL frame having the most number of past selectionscan be considered the most likely selection and can be automaticallychosen by the CPU. This and other methods of choosing frames when userintent is unclear is described in copending parent U.S. patentapplications Ser. Nos. 08/099,861 and 07/889,225, which were previouslyincorporated by reference herein. The returned semantic attribute tagwould refer to the <PERSON> frame for the example shown in FIG. 11.

A <RESTAURANT> type frame 200 is used to trace restaurant names queriedto a database. For example, the CPU would receive the query "Chez Sovan"and can examine a <PLACE> type frame (not shown) to find the<RESTAURANT> type frame 200 in a similar process to that describedabove. The <RESTAURANT> type frame 200 includes the instance frames<RESTAURANT-1> and <RESTAURANT-2>. <RESTAURANT-1> has a field thatmatches the substring "Chez Soyan" and a semantic attribute tagreferring to the <PLACE> frame is returned.

FIG. 12 shows a number of task templates 208 used for implementing atask. The CPU determines whether the frames found in the knowledge basematch at least one of the task templates of frame combinations set forthin FIG. 12. A task is mapped to the mapped substrings using thecollection of semantic attribute tags which identifies the location andtype of substrings in the knowledge base. The substrings are categorizedinto objects according to their type using the returned semanticattribute tags. For example, the substring "lunch with" is considered anaction object of type MEET. Likewise, "Bill" is considered to be aPERSON object, "Chez Sovan" is considered to be a PLACE object, and"Monday 12:30 pm" is considered to be a DAY/TIME object. These fourobjects in combination are subject to template comparison. The templatesshown in FIG. 12 are effective for organizing the various objectcombinations which are capable of further operation as particular tasksto be accomplished. FIG. 12 illustrates selected example tasks 210, suchas scheduling, finding, filing, formatting, mailing, faxing, printing,and calling, just to cite a few of the possibilities.

FIG. 12 further provides example kinds of action objects 212, such asmeet, find, file, format, mail, fax, print, and call. The Figureprovides examples of allowable combinations of objects 214 whichcorrespond to the indicated tasks and actions. For example, essentialobjects for scheduling a meeting include three objects, such as person,place, and day/time. Finding activities require the combination of twoobjects, which are for example a quantifier and an object. Filingrequires a quantifier and notes. Formatting require notes and form,mailing requires a person, a place, and a letter. Faxing requires aperson, a place, a fax number, and notes. Printing requires an objectand a place. Calling requires a person, a place, and a phone number.

A recognition of user intent call; for a matching operation betweenmapped object(s) such as those mapped in FIGS. 9-11 and the objectsexpressed in the templates of FIG. 12. In the example shown, the intentis not obscure. There is a template match with the scheduling taskexpressed in the template. The action object <MEET> of the substring"lunch with" is present in the template for the Scheduling task 210. Theassociated objects 214 of person, place, and day/time are all recognized(mapped) from the input string as well, permitting a complete match tothe Scheduling task. Once a complete match is made, the CPU executes theassociated task 210, which is described in greater detail with referenceto FIG. 13.

Often a complete match to objects in a task template is not possible.Ambiguities may occur if only a few of the objects 214 have beenrecognized, or a set of recognized objects may fit into more than onetask template. The CPU then would have to follow a method for resolvingambiguities in task assignment and user intent. One such method would beto pick the task most frequently selected by the user (or the mostrecent task selected by the user) if at least a partial match betweenthat task and the inputted string has been made. Another such methodwould be to prompt the user with an ambiguity and require the user toclarify the ambiguity. Such methods are discussed in co-pending parentU.S. patent applications Ser. No. 08/099,861 and 07/889,225.

FIG. 13 is a flow diagram illustrating steps 108 and 110 of FIG. 4, inwhich a task is determined based upon the mappings of the substrings andthat task is performed by the CPU. The process begins in a first step220, and, in a step 224, the task template 208 is filled as describedabove with reference to FIG. 12. A task is matched to the mappedsubstrings using the task template. In next step 224, the CPU checks ifthe determined task can be executed. If a task has been matched thatrequires specific parameters or conditions that are not available, thenthe task obviously cannot be executed, and the process is complete atstep 226. Examples of tasks that cannot be executed are a faxing taskwhen no fax machine is connected to the computer, a printing task whenno printer is connected to the computer system, or a task which requiresmore objects or information than have been supplied by the user.

If the task can be executed, the process continues to step 228, whereinthe task is executed by the CPU. Thus, in the example shown from FIG. 3,"Lunch with Bill Monday 12:30 pm at Chez Sovan ", a scheduling task hasbeen matched to the input string and the task is executed. The CPU thenexecutes related software which has been designated for schedulingtasks. For example, the task can cause the CPU to automatically inputthe meeting information in a scheduling calendar program. Or, the taskcan cause the CPU to flash the lunch meeting information in a displayarea of the display screen 20. Once the task has been executed, theprocess is complete as indicated in step 226. As an alternativeadditional step, the CPU can ask for a confirmation from the user beforeimplementing the determined task.

While this invention has been described in terms of several preferredembodiments, it is contemplated that alterations, modifications andpermutations thereof will become apparent to those skilled in the artupon a reading of the specification and study of the drawings.Furthermore, certain terminology has been used for the purposes ofdescriptive clarity, and not to limit the present invention. It istherefore intended that the following appended claims include all suchalterations, modifications and permutations as fall within the truespirit and scope of the present invention.

What is claimed is:
 1. A computer system having the ability to processnatural language, the system comprising:a processor; memory formaintaining a database of task-pertinent events, said memory beingcoupled to said processor; an input device coupled to said processor forreceiving natural language from a user; means for developing a stringfrom said natural language coupled to said input device; means formapping a portion of said string into said database using a plurality ofdifferent mapping routines arranged in a hierarchy, wherein a successivemapping routine in said hierarchy is used when a previous differentmapping routine in said hierarchy does not map said portion of saidstring; and means for performing a computer-implemented task based uponsaid mapped portion of said string.
 2. A computer system as recited inclaim 1 wherein said input device includes a display screen receptive tostylus input.
 3. A computer system as recited in claim 1 wherein saidinput device includes a microphone receptive to speech input.
 4. Acomputer system as recited in claim 1 wherein said plurality ofdifferent mapping routines includes a phrasal mapping routine, a patternmapping routine, and a database query mapping routine.
 5. A computersystem as recited in claim 1 wherein said computer-implemented taskincludes executing an application program stored in said memory.
 6. Acomputer system as recited in claim 1 wherein each of said successivemapping routines in said hierarchy utilizes a different process to mapsaid portion of said string and requires more time to implement by saidprocessor than previous mapping routines in said hierarchy.
 7. Acomputer system as recited in claim 1 wherein said means for mappingincludes a window for selecting said portion of said string mapped intosaid database, wherein said window has a predetermined size and is movedto select other portions of said string when said portion cannot bemapped into said database.
 8. A method for implementing a task input innatural language by a user into a computer system, the methodcomprising:developing a string including recognized word objects frominput received from a user; examining a number of said word objects ofsaid string shown in a word window according to a first mapping routineto map said number of word objects into a database, said word windowhaving a predetermined length; examining said word objects shown in saidword window according to a second mapping routine when said number ofword objects of said string do not map according to said first mappingroutine, said second mapping routine utilizing a different mappingprocess from said first mapping routine and requiring more time toimplement than said first mapping routine when examining said number ofword objects; examining said word objects shown in said word windowaccording to a third mapping routine when said number of word objects ofsaid string do not map according to said second mapping routine, saidthird mapping routine utilizing a different mapping process from saidfirst mapping routine and said second mapping routine and requiring moretime to implement than said second mapping routine when examining saidnumber of word objects; and performing a task in accordance with amapping obtained by said examination of said word objects shown in saidword window.
 9. A method as recited in claim 8 further comprisingexamining different word objects of said string by moving said wordwindow along said string when said number of word objects of said stringdo not map according to said third mapping routine, wherein said lengthof said word window is fixed such that said word window shows apredetermined number of word objects of said string.
 10. A method asrecited in claim 9 wherein said examining different word objectsincludes decreasing said length of said word window such that less wordsare shown in said word window.
 11. A method as recited in claim 8wherein at least one of said mapping routines is a dynamic mappingroutine.
 12. A method as recited in claim 8 wherein said first mappingroutine is a pattern processor utilizing a phrasal look-up table, saidsecond mapping routine is a PATTERN processor utilizing a patternlook-up table, and said third mapping routine is a database queryprocessor.
 13. A method as recited in claim 8 further comprisingexamining additional word objects of said string according to saidmapping routines until all word objects of said string have beenexamined.
 14. A method as recited in claim 8 wherein said third mappingroutine includes a database query processor and wherein said first andsecond mapping routines do not include database query processors.
 15. Amethod for recognizing meanings of natural language phrasescomprising:attempting to map at least a portion of an ordered string ofwords into a database according to a phrasal mapping routine; attemptingto map said portion into a database according to a pattern mappingroutine when said portion does not map according to said phrasal mappingroutine; and attempting to map said portion according to a databasequery mapping routine when said portion does not map according to saidpattern mapping routine.
 16. A method as recited in claim 15 whereinsaid step of attempting to map said portion according to said phrasalmapping routine includes mapping said portion into a phrasal look-uptable.
 17. A method as recited in claim 15 wherein said step ofattempting to map said portion according to said phrasal mapping routineincludes mapping said portion into a pattern look-up table.
 18. A methodas recited in claim 15 further comprising attempting to map remainingportions of said ordered string according to said mapping routines untilall word objects of said string have been attempted to be mapped.
 19. Amethod as recited in claim 18 wherein said remaining portions of saidordered string are attempted to be mapped using a window that is movedto show different portions of said ordered string.
 20. A method asrecited in claim 15 wherein said mapping routines are dynamic mappingroutines.